ENUMERATING PATTERN AVOIDANCE FOR AFFINE PERMUTATIONS 

ANDREW CRITES 

Abstract. In this paper we study pattern avoidance for affine permutations. In particular, we 
show that for a given pattern p, there are only finitely many affine permutations in Sn that avoid 
p if and only if p avoids the pattern 321. We then count the number of affine permutations that 
avoid a given pattern p for each p in 53, as well as give some conjectures for the patterns in S4. 



1. Introduction 

Given a property Q, it is a natural question to ask if there is a simple characterization of all 
permutations with property Q. For example, in [Ifj] the permutations corresponding to smooth 
Schubert varieties are exactly the permutations that avoid the two patterns 3412 and 4231. In 
[li| it was shown that the permutations with Boolean order ideals are exactly the ones that avoid 
the two patterns 321 and 3412. A searchable database listing which classes of permutations avoid 
certain patterns can be found at fl7l ]. 

Since we know pattern avoidance can be used to describe useful classes of permutations, we 
might ask if we can enumerate the permutations avoiding a given pattern or set of patterns. For 
example, in [lil ] it was shown that if S n (p) is the number of permutations in the symmetric group, 
S n , that avoid the pattern p, then there is some constant c such that S n (p) < c n . Thus the rate of 
growth of pattern avoiding permutations is bounded. This result was known as the Stanley- Wilf 
conjecture, now called the Marcus- Tardos Theorem. 

We can express elements of the affine symmetric group, S n , as an infinite sequence of integers, 
and it is still natural to ask if there exists a subsequence with a given relative order. Thus we can 
extend the notion of pattern avoidance to these affine permutations and we can try to count how 
many u G S n avoid a given pattern. 

For p G S m , let 

(1) fP = # G S n : u avoids p\ 

and consider the generating function 

00 

(2) 

n=2 

For a given pattern p there could be infinitely many u G S n that avoid p. In this case, the generating 
function in (J2]) is not even defined. As a first step towards understanding f p (t), we will prove the 
following theorem. 

Theorem 1. Let p G S m . For any n > 2 there exist only finitely many uj G S n that avoid p if and 
only if p avoids the pattern 321. 

It is worth noting that 321-avoiding permutations and 321-avoiding affine permutations appear 
as an interesting class of permutations in their own right. In [2, Theorem 2.1] it was shown that a 
permutation is fully commutative if and only if it is 321-avoiding. This means that every reduced 
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expression may be obtained from any other reduced expression using only relations of the form 
SiSj = SjSi. Moreover, a proof that this result can be extended to affine permutations as well 
appears in [(], Theorem 2.7]. For a detailed discussion of fully commutative elements in other 
Coxeter groups, see [TH ]. 

Even in the case where there might be infinitely many lo € S n that avoid a pattern p, we can 
always construct the following generating function. Let 

(3) 9m,n = # £ S n : lo avoids p and £(lo) = m\ . 
Then set 

oo oo 

(4) gt>(x,y) = Y J Y,9knX m y n - 

n=2 m=0 

Since there are only finitely many elements in Sn of a given length, we always have gm,n < oo. The 
generating function g 321 (x,y) is computed in [7], Theorem 3.2]. 

The outline of this paper is as follows. In Section [2] we will review the definition of the affine 
symmetric group and list several of its useful properties. In Section [3] we will prove Theorem Q3 
which will follow immediately from combining Propositions [Hand [6] In Section [5] we will compute 
f p (t) for all of the patterns in S3. Finally, in Section[5]we will give some basic results and conjectures 
for f p (t) for the patterns in S4. 

2. Background 

Let S n denote of the set of all bijections lo : Z — > Z with co(i + n) = oo(i) + n for all £ G Z and 

<5) S^KT} 

S n is called the affine symmetric group, and the elements of S n are called affine permutations. This 
definition of affine permutations first appeared in [ll|, §3.6] and was then developed in [3]. Note 
that S n also occurs as the affine Weyl group of type A n ~\. 

We can view an affine permutation in its one-line notation as the infinite string of integers 

• • • LU-1LU0LU1 ■ ■ ■ LO n LO n+ i • • • , 

where, for simplicity of notation, we write L0{ = oj(i). An affine permutation is completely deter- 
mined by its action on [n] := {1, . . . , n}. Thus we only need to record the base window [u)i, . . . , Lo n ] 
to capture all of the information about lo. Sometimes however, it will be useful to write down a 
larger section of the one-line notation. 

Given i ^ j mod n, let tij denote the affine transposition that interchanges i + mn and j + mn 
for all m € Z and leaves all k not congruent to i or j fixed. Since t^ = ti +n j +n in S n , it 
suffices to assume 1 < i < n and i < j. Note that if we restrict to the affine permutations with 
{cji, . . . , oj n } = [n], then we get a subgroup of S n isomorphic to S n , the group of permutations of 
[n]. Hence if 1 < i < j < n, the above notion of transposition is the same as for the symmetric 
group. 

Given a permutation p € Sk and an affine permutation w € S n , we say that to avoids the pattern 
p if there is no subsequence of integers i% < ■ ■ ■ < ik such that the subword Wj 1 • • • Ui k of to has the 
same relative order as the elements of p. Otherwise, we say that lo contains p. For example, if 
co = [8,1,3,5,4,0] € Sq, then 8,1,5,0 is an occurrence of the pattern 4231 in lo. However, lo avoids 
the pattern 3412. A pattern can also come from terms outside of the base window [lo\, . . . ,L0 n ]. In 
the previous example, lo also has 2,8,6 as an occurrence of the pattern 132. Choosing a subword 
LOi x ■ ■ ■ LOi k with the same relative order as p will be referred to as placing p in to. 
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2.1. Coxeter Groups. For a general reference on the basics of Coxeter groups, see 0] or [§]. Let 
S = {s±, . . . , s n } be a finite set, and let F denote the free group consisting of all words of finite 
length whose letters come from S. Here the group operation is concatenation of words, so that the 
empty word is the identity element. Let M = (mij)™j = i be any symmetric n x n matrix whose 
entries come from Z>o U {oo} with l's on the diagonal. Then let N be the normal subgroup of F 
generated by the relations 

R = {{ St s d )^ = ir ij=l . 

If rriij = oo, then there is no relationship between Si and Sj. The Coxeter group corresponding to 
S and M is the quotient group W = F/N. 

Any w 6 W can be written as a product of elements from S in infinitely many ways. Every 
such word will be called an expression for w. Any expression of minimal length will be called a 
reduced expression, and the number of letters in such an expression will be denoted £(w), the length 
of w. Call any element of S a simple reflection and any element conjugate to a simple reflection, a 
reflection. 

We graphically encode the relations in a Coxeter group via its Coxeter graph. This is the labeled 
graph whose vertices are the elements of S. We place an edge between two vertices Si and Sj if 
niij > 2 and we label the edge rriij whenever rriij > 3. The Coxeter graphs of all the finite Coxeter 
groups have already been classified. See, for example, @, §2]. 

In 0, §8.3] it was shown that S n is the Coxeter group with generating set S = {so, s\, . 
and relations 



%-i i 



R 



2 



SiSj) = 1, if \i - j\ > 2, 
(sjSj+i) 3 = 1, for < i < n — 1, 



where all of the subscripts are taken mod n. Thus the Coxeter graph for S n is an n-cycle, where 
every edge is unlabeled. 




FIGURE 1. Coxeter graph for S, 



If J C S is a proper subset of S, then we call the subgroup of W generated by just the elements 
of J a parabolic subgroup. Denote this subgroup by Wj. In the case of the affine symmetric group 
we have the following characterization of parabolic subgroups. 

Proposition 2. Let J = S\{si}. Then lo € S n is in the parabolic subgroup (S n )j if and only if 
there exists some integer i<j<i + n — 1 such that ujj < cot < ojj + n for all i < k < i + n — 1. 

Proof. In 0, Proposition 8.3.4] it is shown that (S n )j = Stab([i, i + n — 1]). The result then follows 
from the definition of the stabilizing set. □ 

2.2. Length Function for S n . For oj E S n , let £(uj) denote the length of uj when S n is viewed as 
a Coxeter group. Recall that for a non-affine permutation tt G S n we can define an inversion as a 
pair such that i < j and 7Tj > irj. For an affine permutation, if u>j > ujj for some i < j, then 
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we also have uJi+kn > ^j+kn f° r an k £ *Z. Hence any affine permutation with a single inversion 
has infinitely many inversions. Thus we standardize each inversion as follows. Define an affine 
inversion as a pair (i, j) such that 1 < i < n, i < j, and cu, > ujj. If we let Invg (to) denote the set 
of all affine inversions in ui, then £(uj) = $dnvg (u>), 0, Proposition 8.3.1]. 

We also have the following characterization of the length of an affine permutation, which will be 
useful later. 



Theorem 3. pj, Lemma 4.2.2] Let uj G S n . Then 



l<i<j<n 



(6) t(u)= £ =inv( Wl| ..., WB )+ £ 



l<i<j'<n 



??. 



where inv(wi, . . . ,co n ) = #{1 < i < j < n : > Wj}. 

For 1 < ? < n define Invj(o;) = Gff:i< j,Wj> cjj}. Now let Inv(w) = (Invi(w), . . . , Inv n (w)), 
which will be called the affine inversion table of oj. In [4j, Theorem 4.6] it was shown that there is 
a bijection between S n and elements of Z> containing at least one zero entry. 

3. Proof of Theorem [TJ 

We start with the proof of one direction of Theorem [TJ Proposition [6] will complete the proof. 

Proposition 4. If p G S m contains the pattern 321, then there are infinitely many uj € S n that 
avoid p. 

Proof. For k € N, let u/ fc ) £ »S n be the affine permutation whose reduced expression, when read 
right to left, is obtained as follows. Starting at sq, proceed clockwise around the Coxeter diagram 
k(n — 1) steps, appending each vertex as you go. The base window of the one-line notation of these 
elements has the form 

= [1 - k, 2 - k, . . . , n - 1 - k, n + k(n - 1)]. 

Note these elements correspond with the spiral varieties in the affine Grassmannian from []]]. 
As an example, in S4 we have the following: 

s 2 s lS o = = [0,1,2,7] 

s 1 s s 3 s 2 s 1 so = w (2) = [-1,0,1,10] 
S S3S2S 1 SoS 3 S 2 S 1 So = w (3) = [-2,-1,0,13]. 

The infinite string in the one-line notation of u/ fc ) is a shuffle of two increasing sequences. Hence 
every oj^ avoids the pattern 321. Thus there are infinitely many permutations in S n avoiding the 
pattern 321, and hence avoiding any pattern p containing 321. □ 

Call a permutation p G S m decomposable if p is contained in a proper parabolic subgroup of S m . 
In other words, there exists some 1 < j < m — 1 such that {p\, . . . ,pj} = [j]. Then we have the 
following lemma. 

Lemma 5. Let p € S m be decomposable. Ifp avoids the pattern 321, then there exists some constant 
L such that if £{uj) > L, then oj contains the pattern p. Hence there are only finitely many uj £ S n 
that avoid p. 

Proof. Our proof will use induction on m. If m = 1, the result is clear, since no affine permutations 
can avoid p. So suppose that m > 1 and that for any k < m and q E Sk there exists some constant 
L q such that if £(oj) > L q , then oj contains q. 

Since p is decomposable, there exists an index j such that {p±, . . . ,pj} = Note in this case, 
we also have {pj+i, ■ ■ ■ ,p m } = {j + 1, ■ ■ ■ , m}- So we can view q = p\ ■ ■ -pj as an element of Sj 



ENUMERATING PATTERN AVOIDANCE FOR AFFINE PERMUTATIONS 



5 



and r = Pj+i ■ ■ ■ p m as an element of S m -j. By our induction hypothesis, there exists a constant 
L q such that if £{u) > L q , then there is an occurrence of q in u. Also, there exists a constant L r 
such that if £(u) > L r , then there is an occurrence of r in u. 

Now let L = max{Lg,L r } and suppose l{u) > L. Then u contains both patterns q and r. By 
the periodic property of u, we can translate the occurrence of r to the right some multiple of n 
until it lies entirely to the right of the occurrence of q and the smallest entry of r is bigger than the 
largest entry of q. This will now give an occurrence of p in u. More specifically, let u ai ■ ■ ■ u aj be 
the occurrence of q in u, and let u bl ■ ■ ■ ^b^-j be the occurrence of r in uj. Then there exists some 
k £ N such that b\ + kn > a,j and uj at < ut> s+kn f° r every 1 < t < j and 1 < s < m — j. We then 
have u ai ■ ■ ■ u a .u bl+kn ■ ■ ■ uj bm _. +kn is an occurrence of p in uj. 

Hence if £(uj) > L, then uj must contain p. Thus there can only be finitely many uj € S n that 
avoid p since #{w € S n : £(uj) < L} is finite. □ 

We now want to prove a similar statement for a general 321-avoiding pattern. 



Proposition 6. If p € 5„ 

avoid p. 



avoids the pattern 321, then there are only finitely many uj € S n that 



Proof. Suppose p avoids the pattern 321. By Lemma [5] we may also assume p is indecomposable. 
Let a = a\ ■ ■ ■ be the subsequence of p consisting of all pj such that pi < pj for all i < j. Here a 
is just the sequence of left-to-right maxima. Let b be the subsequence of p consisting of all pi not in 
a. By its construction, a must be increasing. Furthermore, since p avoids the pattern 321, b must 
also be increasing. To see this, note that if there is some p s ,Pt m b with s < t and p s > pt, then 
there is some r < s with p r > p s , since p s is not in a. But then p r p s pt forms a 321 pattern in p. 
Let uj G S n and suppose that for some 1 < a < f3 < n, we have 

\ojp - UJ a 



n 



> m £+1 + 1. 



If uo a < up, set uo' a = uip and uj'a 



UJ, 



+ n. Then we will have uj' n > uj'^ and 



\UJ.-_ 



UJ' 



n 



> m 



£+1 



So in what follows we will assume uj a > ujp and 
(T) ^ 



UJ,, 



)1 



> TO 



e+i 



We can now construct the occurrence of p in uj. Our iterative algorithm will complete in I steps, 
where £ is the length of the subsequence a described above. We will be using translates uj a+ k n 
to place the terms of p in the a sequence and translates ujp+kn to place the terms of p in the b 
sequence. 

Since p is indecomposable, a\ ^ 1. Hence there is some t such that b t = a\ — 1. Suppose b t = Pi. 
Let s be the largest index such that a s lies to the left of bt in p. Note that 1 < s < to or else 
p is decomposable. Let y be the largest integer such that ujp +yn < uj a and let z = [|J . Since 
uo a — ujp > nm e+1 , we have y > m e+1 and hence z > m e . For each 1 < k < s, use u a +(k-i)zn to 
place cifc in uj. Then if ui u corresponds to a k and uj v corresponds to at+i, we will have 

(8) \uj u — uj v \ = \u — v\ = nz > nm . 

Finally, use translates of uip to place b\, ... ,b t in uj in such a way that b t is placed at uip +yn and for 
any 1 < x < t, if b x lies between a k and a^+i in p, then 6^ is placed at a translate of uip between 
a; o+(fe-i)zn an d w a+ fc2„. By ([8]) there are at least m i translates of up in this interval, so there is 
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0-2 
• 



b 2 



Figure 2. First place all values of p to the left of b t . 



a r+ i 



. • Pi 



h =Pi 



Figure 3. The (r + l) st iteration will place all elements of p between pi + \ and pj. 



enough space to place all of the b x 's that lie between and a^+i using translates of wg. Thus after 
the first iteration we have placed p\ - • - pi in uj. 

Now suppose we have placed every term in the a sequence up to a r for some 1 < r < £. If we have 
placed a r , then we have also placed some additional terms from the b sequence. Again, fix t so that 
bt is the largest element in p to the right of a r satisfying bt < a r . We may assume such a bt exists, 
or else p is decomposable. If b t = pi, then we have actually placed p\ ■ ■ - pi- Moreover, suppose that 
the terms from the a sequence among p\ - ■ ■ Pi have been placed so that if ui u corresponds to a/% and 
oj v corresponds to dk+i for some 1 < k < r, then 



(9) 



\w u — uj v \ = \u — v > nm 



e-r+i 



Note we must have also already placed a r+ i, or else a r+ \ = pi + \ and hence p is decomposable. 

We will now show how to place all terms in p from the b sequence whose values are between a r 
and a r +i, thus completing the (r + l) st step of our algorithm. Note that in the process of placing 
these terms, we will also possibly be placing some additional terms from the a sequence. Let oj u 
correspond to a r and u v correspond to a r+ \. Then we have at least m l ~ r+1 translates of uj a and ujp 
falling between ui u and w v . So if pj is the largest entry of p to the left of a r+ \ satisfying pj < a r+ i, 
as in the first step of our algorithm, we may place p%+i, ■ ■ ■ ,Pj in such a way that any of the terms 
corresponding to the subsequence a are placed at least m e ~ r translates apart. 
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Iterating this algorithm £ times will place all of p in uj. Hence if uj is to avoid p, then we must 
have 

\uj/3 - UJ C 



n 



< m e+1 + 1 for all 1 < a < (3 < n. 



Since inv(u;i, . . . , uj n ) < f,) , we conclude by ([6]) that 



(10) £{u) < ^ j + (m w + lj ^ j = (m _ t v o 

For any fc, the set of all affine permutations in S n of length at most k is finite. Hence there can be 
only finitely many elements in S n that avoid p. □ 

Note that in general, the length bound £{uj) < (m^ +1 + 2) (g) is much larger than needed. For 
the proof of Theorem [1] though, any upper bound on £(uj) will suffice. Given a specific pattern p, 
we can tighten the bounds in the above algorithm, and thus obtain better upper bounds on the 
maximal length for pattern avoidance. 

For example, let p = 3412 € S4. By (fT0|) . if oj G S n avoids p, then £(cj) < 66Q). Here the 
algorithm is completed on the first iteration and we can actually prove a tighter bound £{uS) < 3Q) 
for this particular pattern. 

4. Generating Functions for Patterns in S 3 

Let fn and f p (t) be as in (pQ) and ([2]) in Section [TJ Then by Theorem [T] we have /^ 21 = 00 for all 
n. However, for all of the other patterns p £ S3 we can still compute f p (t). 

Theorem 7. Let f p (t) be as above. Then 

(11) f 123 (t) = 0, 

00 

(12) / 132 W = / 213 (t) = X>, 

n=2 



(13) / 231 (i) = / 312 (*) = £ 



2n 



n=2 



To make the proof easier, we first study a few operations on S n that interact with pattern 
avoidance in a predictable way. 

Lemma 8. Let uj € S n and p € S m . Then uj avoids p if and only i/w -1 avoids p^ 1 . 



Proof. The proof is the same as the one for non-affine permutations given in 18|, Lemma 1.2.4]. 
Suppose uj contains p, so that uj^uj^ ■ ■ ■ uii m is an occurrence of p in uj. Let jk = for 1 < k < m. 
Then uij 1 ■ ■ ■ uij 1 will give an occurrence of p~ x in uj^ 1 . □ 

31 Jm u 

Now define a map o T : S n — >■ S n by setting 

I + 1, if 2 < i < n, 



a r (uj)i 



ui n — n + 1, if i = 1. 



This has the effect of shifting the base window of uj one space to the right, while preserving the 
relative order of the elements. The affine inversion table of o~ r (cu) is a barrel shift of the affine 
inversion table of uj one space to the right. Similarly, define o~i = u" 1 , which will perform a barrel 
shift one space to the left. Thus a r is the length-preserving automorphism of S n of order n obtained 
by rotating the Coxeter graph one space clockwise. 
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For example, if uj = [5, —4,6,3] € S4, which has affine inversion table (4,0,3, 1), then a r (co) = 
[0, 6, —3, 7], which has affine inversion table (1, 4, 0, 3). 

Lemma 9. Let co € S n and p £ S m . The following are equivalent. 

(1) co avoids p. 

(2) a r (co) avoids p. 

(3) o-£(oj) avoids p. 

Proof. The relative order of elements in co is unchanged after applying a r or og. Hence if co^ ■ ■ ■ coi m 
is an occurrence of p in co, then co il+ i • • • w^+i is an occurrence of p in a T (co) and w^-i • • • ^i m -i is 
an occurrence of p in ai(co). □ 

We are now ready to enumerate the affine permutations that avoid a given pattern in S3. 

Proof of Theorem^ For any co G S n , the entries coicoi +n coi + 2n are always an occurrence of 123 in 
co. Hence /^ 23 = for all n. If co has a descent at co,i so that co^ > Wj+j, then there is some translate 
i — sn such that coi- sn < Wj+i. Hence coi- sn coiCOi + i is an occurrence of 132 in co. Also, coi +n > oJi+x 
so that C0ico i+ ic0i +n is an occurrence of 213 in co. Thus the only affine permutation that can avoid 
132 or 213 is the identity. Hence /* 32 = / 213 = 1. 

By Lemma [8] we have / 231 = / 312 . Thus it remains to compute /„ . So suppose w avoids 231. 
We first show a; is in a proper parabolic subgroup that depends on the position and value of the 
maximal element of the base window. 

Let a be the index such that co a = maxjwi, . . . , co n }. First suppose co a > n + a — 1. Shift co to 
the left a — 1 times, setting v = a^~ l {co). Then v\ = co a — a + 1 > n. Since v must satisfy (J5]), 
there must exist some 1 < j < n with Vj < 0. Then ^i_ n ^i^ is an occurrence of 231 in v. By 
Lemma [9l co contains 231, which is a contradiction. So we must have n < co a < n + a — 1. 

Now let u = a^ a ~ n {co). Set i = a — co a + n so that Ui = n. If {u±, . . . , u n } ^ [n], then since u 
must satisfy ([5]), there is some 1 < j,k < n such that uj < and > n. Since co a was chosen to 
be maximal, we must have i < k. Then UiUkUj+ n will give an occurrence of 231 in u and hence 
also in co by Lemma [9j giving a contradiction. Hence u € S n C 5^. 

Let C n = ^Tj- ( 2 J [ l ) be the n th Catalan number. Recall from [9| that there are C n 231-avoiding 
permutations in S n . Again, suppose co a = maxjwi, . . . , co n } and co a = n + a — i, for some 1 < i < a. 
Then u = a^ a ~ n (co) is an element in S n with m = n. Furthermore, we have < Uj for every pair 
h < i < j. There are Ci-\C n -i such permutations. Summing over all possible values of i gives 



many 231-avoiding affine permutations whose maximal value in the base window occurs at index 
a. Summing over all 1 < a < n then gives 



a a—1 



i=l i=0 



(14) 




Using the defining recurrence, 



n-l 



(15) 




for the Catalan numbers, (|14p simplifies to 



(16) 
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Conversely, if u S S n C S n is a 231-avoiding permutation with U{ = n, then a 3 r {u) will be a 
231-avoiding affine permutation for any < j < n — i. Thus we actually have equality in (|16p . 
completing the proof. □ 



5. Generating Functions for Patterns in S 4 

We now look at pattern avoidance for patterns in S4. There are 24 patterns to consider, although 
for all but three patterns, f p (t) is easy to compute. First let 

P = {1432, 2431, 3214, 3241, 3421, 4132, 4213, 4231, 4312, 4321}. 

By Theorem (TJ if p £ P, then f% = 00, so f p (t) is not defined. 

Theorem 10. We have 

(17) / 1234 (i) = 0, 

00 

(18) / 1243 (i) = / 1324 (t) = / 2134 (t) = f 2U \t) = £ t» 

n=2 

(19) / 1342 (t) = / 1423 (t) = / 2314 (o = f 12 Ht) = f; ( 2n " X V 

n=2 V n / 

Proof. As in Theorem [7] there are no affine permutations avoiding 1234, and only the identity 
permutation avoids 1243, 1324, 2134, or 2143. If w^cj^oj^ is an occurrence of 231 in u, then there 
is some translate i\ — sn such that LOi l — sn <C ^23. Hence sn^i\^i2^iz 1^ occurrence of 1342 in 
a;. Conversely, if oj avoids 231, then it must also avoid any pattern containing 231, namely 1342. 
This shows Z 1342 = / 231 . Similarly, we also have J 1423 = / 2314 = / 3124 = / 231 . □ 

Based on some initial calculations, we also have the following conjectures for the remaining three 
patterns in S^. 

Conjecture 1. The following equalities hold. 

(20) f^ 2 = n t { ^{ n ~i +k y 

k=o n \ J 

Pi) * a = iS" = \t§(5 



Note that ([20D is sequence A064062 and dH} is sequence A087457 in [14]]. It is also worth 
comparing (|21[) to the number of 3412-avoiding, non-affine permutations given in [5], §7] as 

A [n\ 2 (2k\ 3k 2 + 2k + l-n-2kn 
(22) U ^ = 2 2^{ k ) {k)(k+ink + 2)(n-k + iy 
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